1. Prior Art on out-of-position occupants and rear facing child seats
Whereas thousands of lives have been saved by airbags, a large number of people have also been injured, some seriously, by the deploying airbag, and thus significant improvements need to be made in this regard. As discussed in detail in one or more of the copending patent applications cross-referenced above, for a variety of reasons vehicle occupants may be too close to the airbag before it deploys and can be seriously injured or killed as a result of the deployment thereof. Also, a child in a rear facing child seat which is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as first publicly disclosed in Breed, D. S. "How Airbags Work" presented at the International Conference on Seatbelts and Airbags in 1993, in Canada, occupant position sensing and rear facing child seat detection is required in order to minimize the damages caused by deploying airbags.
Initially these systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded and unwanted airbag deployments when a front seat is unoccupied. However, airbags are now under development to protect rear seat occupants in vehicle crashes and all occupants in side impacts. A system will therefore be needed to detect the presence of occupants, determine if they are out-of-position and to identify the presence of a rear facing child seat in the rear seat. Future automobiles are expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance and possible harm of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident needlessly.
Inflators now exist which will adjust the amount of gas flowing to the airbag to account for the size and position of the occupant and for the severity of the accident, e.g., based on the rate of flow of the inflating gas. The vehicle identification and monitoring system (VIMS) discussed in co-pending application Ser. No. 08/239,978, now abandoned, among others, will control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. The instant invention is an improvement on that VIMS system and uses an advanced ultrasonic system comprising two or more ultrasonic transmitters/receivers combined with a trained neural network pattern recognition system as discussed in much greater detail below.
The automatic adjustment of the deployment rate of the airbag based on occupant identification and position and on crash severity has been termed "smart airbags". Central to the development of smart airbags is the occupant identification and position system described herein. To complete the development, an anticipatory crash detecting system such as disclosed in U.S. patent application Ser. No. 08/247,760 now abandoned, filed May 23, 1994 is desirable. Prior to the implementation of anticipatory crash sensing, the use of a neural network smart crash sensor which identifies the type of crash and thus its severity based on the early part of the crash acceleration signature should be developed and thereafter implemented. U.S. patent application Ser. No. 08/476,076 now U.S. Pat. No. 5,684,701, filed Jun. 7, 1995 describes a crash sensor based on neural networks. This crash sensor, as with all other crash sensors, determines whether or not the crash is of sufficient severity to require deployment of the airbag and, if so, initiates the deployment. A neural network based on a smart airbag crash sensor could also be designed to identify the crash and categorize it with regard to severity thus permitting the airbag deployment to be matched not only to the characteristics and position of the occupant but also the severity and timing of the crash itself
The need for an occupant out-of-position sensor has also been observed by others and several methods have been disclosed in certain U.S. patents for determining the position of an occupant of a motor vehicle. Each of these systems will be discussed below and unfortunately have significant limitations.
In White et al. (U.S. Pat. No. 5,071,160), for example, a single acoustic sensor and detector is described and, as illustrated, is mounted lower than the steering wheel. White et al. correctly perceive that such a sensor could be defeated, and the airbag falsely deployed, by an occupant adjusting the control knobs on the radio and thus they suggest the use of a plurality of such sensors but do not disclose where they would be mounted, other than on the instrument panel below the steering wheel, or how they would be combined to uniquely monitor particular locations in the passenger compartment and to identify what is occupying those locations.
Mattes et al. (U.S. Pat. No. 5,118,134) describe a variety of methods of measuring the change in position of an occupant including ultrasonic, active or passive infrared and microwave radar sensors, and an electric eye. Their use of these sensors is to measure the change in position of an occupant during a crash and use that information to assess the severity of the crash and thereby decide whether or not to deploy the airbag. They are thus using the occupant motion as a crash sensor. No mention is made of determining the out-of-position status of the occupant or of any of the other features of occupant monitoring as disclosed in the above cross-referenced patent applications. It is interesting to note that nowhere does Mattes et al. discuss how to use a combination of ultrasonic sensors/transmitters to identify the presence of a human occupant and then to find his/her location in the passenger compartment.
The object of an occupant out-of-position sensor is to determine the location of the head and/or chest of the vehicle occupant relative to the airbag since it is the impact of either the head or chest with the deploying airbag which can result in serious injuries. Both White et al. and Mattes et al. disclose only lower mounting locations of their sensors which are mounted in front of the occupant such as on the dashboard or below the steering wheel. Both such mounting locations are particularly prone to detection errors due to positioning of the occupant's hands, arms and legs. This would require at least three, and preferably more, such sensors and detectors and an appropriate logic circuitry which ignores readings from some sensors if such readings are inconsistent with others, for the case, for example, where the driver's arms are the closest objects to two of the sensors.
White et al. also describe the use of error correction circuitry, without defining or illustrating the circuitry, to differentiate between the velocity of one of the occupant's hands, as in the case where he/she is adjusting the knob on the radio, and the remainder of the occupant. Three ultrasonic sensors of the type disclosed by White et al. might, in some cases, accomplish this differentiation if two of them indicated that the occupant was not moving while the third was indicating that he or she was moving. Such a combination, however, would not differentiate between an occupant with both hands and arms in the path of the ultrasonic transmitter at such a location that they were blocking a substantial view of the occupant's head or chest. Since the sizes and driving positions of occupants are extremely varied, trained pattern recognition systems, such as neural networks, are required when a clear view of the occupant, unimpeded by his/her extremities, cannot be guaranteed. White et al. do not suggest the use of such neural networks.
Fujita et al., in U.S. Pat. No. 5,074,583, describe another method of determining the position of the occupant but do not use this information to suppress deployment if the occupant is out-of-position, or if a rear facing child seat is present. In fact the closer that the occupant gets to the airbag the faster the inflation rate of the airbag is according to the Fujita patent, which thereby increases the possibility of injuring the occupant. Fujita et al. do not measure the occupant directly but instead determine his or her position indirectly from measurements of the seat position and the vertical size of the occupant relative to the seat. This occupant height is determined using an ultrasonic displacement sensor mounted directly above the occupant's head.
It is important to note that in all cases in the prior art, except those assigned to the current assignee of the instant invention, where ultrasonic sensors are used to determine displacement, only the initial return of reflected waves is used so that only the distance to the closest part of the object can be determined. In contrast, in the instant invention, the return echo pattern over several milliseconds corresponding to the entire portion of the passenger compartment volume of interest is analyzed providing distance information to many points on the items occupying the passenger compartment
2. Definitions
The use of pattern recognition is central to the instant invention as well as those cross-referenced patent applications above. Nowhere in the prior art, except in that assigned to the current assignee of the instant invention, is pattern recognition which is based on training, as exemplified through the use of neural networks, mentioned for use in monitoring the interior passenger compartment or exterior environments of the vehicle.
"Pattern recognition" as used herein will mean any system which processes a signal that is generated by an object, e.g., representative of a pattern of returned or received impulses, waves or other physical property specific to and/or characteristic of and/or representative of that object, or is modified by interacting with an object, in order to determine to which one of a set of classes that the object belongs. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally a series of electrical signals coming from transducers which are sensitive to acoustic (ultrasonic) radiation.
A trainable or a trained pattern recognition system as used herein means a pattern recognition system which is taught to recognize various patterns constituted within the signals by subjecting the system to a variety of examples. The most successful such system is the neural network. Thus, to generate the pattern recognition algorithm, test data is first obtained which constitutes a plurality of sets of returned waves, or wave patterns, from an object and an indication of the identify of that object, i.e., a number of different objects are tested to obtain the unique wave patterns from each object. As such the algorithm is generated, and stored in a computer processor, and which can later be applied to provide the identity of an object based on the wave pattern being received during use by a receiver connected to the processor. For the purposes, the identity of an object sometimes applies to not only the object itself but also to its location in the passenger compartment. For example, a rear facing child seat is a different object than a forward facing child seat and an out-of-position adult is a different object than a normally seated adult.
To "identify" as used herein will mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all rear facing child seats, one containing all human occupants, or all human occupants not sitting in a rear facing child seat depending on the purpose of the system. In the case where a particular person is to be recognized, the set or class will contain only a single element, i.e., the person to be recognized.
An "occupying item" of a seat may be a living occupant such as a human being or a dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
"Out-of-position" as used for an occupant means that the occupant, either driver or passenger, is sufficiently close to the airbag prior to deployment that he or she is likely to be more seriously injured by the deployment event itself than by the accident. This typically occurs when the occupant's head or chest is closer than some distance such as about 5 inches from the deployment door of the airbag module. The actual distance value where airbag deployment should be suppressed depends on the design of the airbag module and is typically further for the passenger airbag than for the driver airbag.
"Transducer" as used herein will in general mean the combination of a transmitter and a receiver. In come cases, the same device will serve both as the transmitter and receiver while in others two separate devices adjacent to each other will be used.
In the description herein on anticipatory sensing, the term "approaching" when used in connection with the mention of an object or vehicle approaching another will mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact with a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The "target" vehicle is the vehicle which is being impacted. This convention permits a general description to cover all of the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
3. Pattern recognition prior art
Japanese patent 3-42337 (A) to Ueno discloses a device for detecting the driving condition of a vehicle driver comprising a light emitter for irradiating the face of the driver and a means for picking up the image of the driver and storing it for later analysis. Means are provided for locating the eyes of the driver and then the irises of the eyes and then determining if the driver is looking to the side or sleeping. Ueno determines the state of the eyes of the occupant rather than determining the location of the eyes relative to the other parts of the vehicle passenger compartment. Such a system can be defeated if the driver is wearing glasses, particularly sunglasses, or another optical device which obstructs a clear view of his/her eyes. Pattern recognition technologies such as neural networks are not used.
U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules to isolate the eyes and mouth of a driver and uses this information to permit the driver to control the radio, for example, or other systems within the vehicle by moving his eyes and/or mouth. Ando uses natural light and illuminates only the head of the driver. He also makes no use of trainable pattern recognition systems such as neural networks, nor is there any attempt to identify the contents of the vehicle nor of their location relative to the vehicle passenger compartment. Rather, Ando is limited to control of vehicle devices by responding to motion of the driver's mouth and eyes.
U.S. Pat. No. 5,298,732 to Chen also concentrates in locating the eyes of the driver so as to position a light filter between a light source such as the sun or the lights of an oncoming vehicle, and the driver's eyes. Chen does not explain in detail how the eyes are located but does supply a calibration system whereby the driver can adjust the filter so that it is at the proper position relative to his or her eyes. Chen references the use of an automatic equipment for determining the location of the eyes but does not describe how this equipment works. In any event, there is no mention of monitoring the position of the occupant, other that the eyes, of determining the position of the eyes relative to the passenger compartment, or of identifying any other object in the vehicle other than the driver's eyes. Also, there is no mention of the use of a trainable pattern recognition system.
U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle. Faris locates the eyes of the occupant by the use of two spaced apart infrared cameras using passive infrared radiation from the eyes of the driver. Again, Faris is only interested in locating the driver's eyes relative to the sun or oncoming headlights and does not identify or monitor the occupant or locate the occupant, a rear facing child seat or any other object for that matter, relative to the passenger compartment or the airbag. Also, Faris does not use trainable pattern recognition techniques such as neural networks. Faris, in fact, does not even say how the eyes of the occupant are located but refers the reader to a book entitled Robot Vision (1991) by Berthold Horn, published by MIT Press, Cambridge, Mass. Also, Faris uses the passive infrared radiation rather than illuminating the occupant with ultrasonic radiation as in the instant invention.
The use of neural networks as the pattern recognition technology is central to this invention since it makes the monitoring system robust, reliable and practical. The resulting algorithm created by the neural network program is usually only a few lines of code written in the C computer language as opposed to typically hundreds of lines when the techniques of the above patents to Ando, Chen and Faris are implemented. As a result, the resulting systems are easy to implement at a low cost making them practical for automotive applications. The cost of the ultrasonic transducers, for example, is expected to be less than about $1 in automotive quantities. Similarly, the implementation of the techniques of the above referenced patents requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than about $5.
The present invention uses sophisticated trainable pattern recognition capabilities such as neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction techniques. A non-automotive example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. "Analysis of Hidden Units in a Layered Network Trained to Classify Sonar Targets", Neural Networks, Vol. 1. pp. 75-89, 1988, and "Learned Classification of Sonar Targets Using a Massively Parallel Network", IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled "Process and Apparatus for the Automatic Detection and Extraction of Features in Images and Displays" to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled "Artificial Neuron and Method of Using Same and U.S. patent application Ser. No. 08/076,601 entitled "Neural Network and Method of Using Same" to Wang, S. T. Other examples include U.S. Pat. Nos. 5,235,339 (Morrison et al.), 5,214,744 (Schweizer et al), 5,181,254 (Schweizer et al), and 4,881,270 (Knecht et al). All of the above references are included herein by reference.
4. Ultrasonics
Ultrasonics can be used in several configurations for monitoring the interior of a passenger compartment of an automobile as described in the cross referenced patents and patent applications. In one known system, for example, two ultrasonic sensors are placed on the A-pillar and in another system, a third sensor is additionally placed in the headliner. It has been found in both of these cases that even though the proper identification is made in a high percentage of the cases, there are still a small but significant number of cases where an error in diagnosis is made based on the information received from the sensors. These systems, although a significant improvement over the other prior art, still fail to achieve the very high reliability desired by the automobile manufacturers.
In the cases of the instant invention as will discussed in more detail below, regardless of the number of transducers used, a trained pattern recognition system, as defined above, is used to identify and classify, and in some cases to locate, the illuminated object and its constituent parts.
5. Applications
The applications for this technology are numerous as described in the copending patent applications listed above. However, the main focus of the instant invention is for the detection of the presence of a child seat in the rear facing position or an out-of-position occupant and the detection of an occupant in a normal seating position. In the former two cases, deployment of the airbag will be suppressed and in the latter, it will be enabled.